The invention relates generally to the field of digital image processing and more particularly to the field of mosaicing systems and methods for producing a single image from plurality of overlapping images.
Mosaicing and super resolution are two methodologies for combining information from a plurality of frames of image data into a single frame. By use of mosaicing, a single panorama mosaic image (hereinafter, generally a xe2x80x9cmosaic imagexe2x80x9d) can be generated from a sequence of image data frames taken by, for example, a video camera that has been panned across a scene or otherwise translated over the scene. The resulting mosaic image can, in a single image, provide a more extensive view of the scene than would typically be possible in a single image recorded by a camera with a normal lens, and would avoid distortions which are common in wide-angle and especially so-called xe2x80x9cfish-eyexe2x80x9d lenses. Methods of generating a mosaic image from a sequence of video images are described in, for example, M. Irani, et al., xe2x80x9cMosaic Based Representations Of Video Sequences And Their Applications,xe2x80x9d Fifth Int""l Conf. on Computer Vision, 1995, pp. 605-611 [hereinafter xe2x80x9cIrani(1)xe2x80x9d], and in S. Peleg, et al., xe2x80x9cPanoramic Mosaics By Manifold Projection,xe2x80x9d IEEE Conf. On Computer Vision And Pattern Recognition, 1997, pp 338-343 [hereinafter xe2x80x9cPelegxe2x80x9d].
Generally, mosaicing involves several phases, including an alignment phase and an integration phase. In the alignment phase, information from overlapping regions of pairs of images is used to determine respective transformations which best align the respective pairs of images. In particular, in each image features are located relative to a coordinate system associated with the respective image, and in alignment the geometric transformations are determined which relate the coordinates of the features in the overlapping region of one image to the coordinate system of the other image, or to a common coordinate system. After the transformations have been determined for the images to be used in generating the mosaic image, the mosaic image can be generated during the integration phase. In the integration phase, the images are integrated into a single mosaic image using the transformation information generated during the alignment phase. In that operation, several methodologies can be used to integrate the overlapping regions of respective images into a single image in the mosaic. In one methodology, the overlapping region from only one image is used in the mosaic, and the corresponding overlapping regions in the other image or images are ignored. In another methodology, the overlapping regions from all of the images are combined in the mosaic. In the second methodology, the contributions from the overlapping regions of the images may be averaged (that is, weighted equally), or they may be weighted according to a selected weighting scheme. As the images are combined into the mosaic, they may be processed to, for example, provide continuity in the illumination level as among the portions provided by the individual images.
The accuracy of the alignment as among the respective images is important in the quality of the mosaic. Alignment accuracy depends on a number of factors, including the relative motion of the camera and objects in the scene, lens distortions, the three-dimensional structure of the scene and noise. When, for example, an overlapping region is taken only from a single image, as described in Peleg, alignment is mostly important along the seams between regions taken from different images. In many cases, for example, rigid translational and rotational transformations for the overlapping region of the respective pairs of images are sufficient to give good alignment along the seams. But alignment limited to rigid transformations may be less accurate in overlapping regions which are not on the seam. Even though rigid transformations are not the most accurate, they are commonly used as they reduce the amount of time required to compute the alignment during the alignment phase, and can lead to reduced global distortions in the mosaic.
By use of super-resolution techniques, information from overlapping regions of multiple image frames can be used to improve the resolution of, and reduce noise in, a resultant image frame. In one super-resolution methodology, described in M. Irani, et al., xe2x80x9cMotion Analysis For Image Enhancement Resolution, Occlusion, And Transparency,xe2x80x9d Jour. Visual Communication And Image Representation, Vol. 4, No. 4, Dec., 1993, pp. 324-335, particularly section 3 thereof [hereinafter xe2x80x9cIrani(2)xe2x80x9d], a super-resolution methodology is described in which, starting with an initial guess as to an appropriate super-resolution image of a scene, a plurality of low-resolution images are generated and compared to actual low-resolution images which were recorded of the scene. By determining differences between the generated and actual low-resolution images, an error function is developed which is used in updating the initial guess super-resolution image. This process can be repeated through a series of iterations until a final super-resolution image is generated with enhanced resolution over the actual low-resolution images.
A problem arises in connection with use of super-resolution techniques for mosaic images, particularly in connection with the use of information from overlapping regions to try to enhance resolution of the corresponding regions of mosaic images. In particular, alignment accuracy over overlapping regions is very important for super-resolution. Super-resolution techniques can be used to improve resolution if the alignment is accurate. However, if alignment is not accurate, super-resolution techniques can serve to degrade resolution in the mosaic image.
The invention provides a new and improved system and method for generating a super-resolution-enhanced mosaic image, using mosaicing and super-resolution methodologies, which provides for enhanced resolution of a mosaic image even if the alignment of the images used in generating the mosaic image is not accurate enough across the overlapping regions to be used directly for super-resolution.
In brief summary, the invention provides a super-resolution-enhanced mosaic image generating system for generating a super-resolution-enhanced mosaic image from a plurality of image frames, each image frame being associated with image data representative of an image of a portion of a scene. The mosaic image generating system comprises a mosaic image generator and a super-resolution generator. The mosaic image generator uses the image data from the image frames to generate mosaic image data representing a mosaic image. The super-resolution generator uses the mosaic image data and the image data from the image frames to generate the super-resolution-enhanced mosaic image. In that process, the super-resolution generator divides the mosaic image into a plurality of patches, each patch associated with at least one image frame, and performing a super-resolution operation in connection with the patch to generate the super-resolution-enhanced mosaic image.